System and method of facilitating cooling of electronics racks of a data center employing multiple cooling stations

ABSTRACT

A cooling system and method are provided for cooling air exiting one or more electronics racks of a data center. The cooling system includes at least one cooling station separate and freestanding from at least one respective electronics rack of the data center, and configured for disposition of an air outlet side of electronics rack adjacent thereto for cooling egressing air from the electronics rack. The cooling station includes a frame structure separate and freestanding from the respective electronics rack, and an air-to-liquid heat exchange assembly supported by the frame structure. The heat exchange assembly includes an inlet and an outlet configured to respectively couple to coolant supply and coolant return lines for facilitating passage of coolant therethrough. The air-to-liquid heat exchange assembly is sized to cool egressing air from the air outlet side of the respective electronics rack before being expelled into the data center.

TECHNICAL FIELD OF THE INVENTION

The present invention relates in general to systems and methods forfacilitating cooling of rack-mounted assemblages of individualelectronics units, such as rack-mounted computer server units.

BACKGROUND OF THE INVENTION

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system level. Increased air flow rates are neededto effectively cool high power modules and to limit the temperature ofthe air that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power, etc.) arepackaged in removable drawer configurations stacked within a rack orframe. In other cases, the electronics may be in fixed locations withinthe rack or frame. Typically, the components are cooled by air moving inparallel air flow paths, usually front-to-back, impelled by one or moreair moving devices (e.g., fans or blowers). In some cases it may bepossible to handle increased power dissipation within a single drawer byproviding greater air flow, through the use of a more powerful airmoving device or by increasing the rotational speed (i.e., RPMs) of anexisting air moving device. However, this approach is becomingproblematic at the rack level in the context of a computer installation(i.e., data center).

The sensible heat load carried by the air exiting the rack is stressingthe ability of the room air-conditioning to effectively handle the load.This is especially true for large installations with “server farms” orlarge banks of computer racks close together. In such installations notonly will the room air-conditioning be challenged, but the situation mayalso result in recirculation problems with some fraction of the “hot”air exiting one rack unit being drawn into the air inlet of the samerack or a nearby rack. This recirculating flow is often extremelycomplex in nature, and can lead to significantly higher rack inlettemperatures than expected. This increase in cooling air temperature mayresult in components exceeding their allowable operating temperature andin a reduction in long term reliability of the components.

Thus, there is a need in the art for a system and method for furtherfacilitating cooling of rack-mounted electronics units, particularly ina large computer room installation traditionally relying on one or moreroom air-conditioning units to cool the electronics racks.

SUMMARY OF THE INVENTION

The shortcomings of the prior art overcome and additional advantages areprovided through the provision of a cooling system for facilitatingcooling of an electronics rack. The cooling system includes at least onecooling station for cooling at least one electronics rack of a datacenter. Each electronics rack of the at least one electronics rackincludes an air inlet side and an air outlet side. The air inlet and airoutlet sides of the electronics rack respectively enable ingress andegress of external air. Each cooling station of the at least one coolingstation is separate and freestanding from at least one respectiveelectronics rack when the at least one electronics rack is in operativeposition adjacent thereto, and is configured for disposition of the airoutlet side of the at least one respective electronics rack adjacentthereto for cooling egressing air therefrom. The cooling stationincludes a frame structure separate and freestanding from the at leastone respective electronics rack; an air-to-liquid heat exchange assemblysupported by the frame structure, the air-to-liquid heat exchangeassembly including an inlet configured to couple to a coolant supplyline and an outlet configured to couple to a coolant return line tofacilitate passage of coolant there through; and wherein theair-to-liquid heat exchange assembly is sized to cool egressing air fromthe air outlet side of the at least one respective electronics rackbefore being expelled into the data center.

In another aspect, a data center is provided which includes multipleelectronics racks and a cooling system for cooling the multipleelectronics racks. Each electronics rack includes an air inlet side andan air outlet side, with the air inlet and air outlet sides respectivelyenabling ingress and egress of external air. The cooling system includesmultiple cooling stations, which are separate and freestanding from themultiple electronics racks. The multiple electronics racks are disposedwith the air outlet sides thereof adjacent to the multiple coolingstations for facilitating cooling of egressing air therefrom. Eachcooling station further includes a frame structure separate andfreestanding from at least one respective electronics rack, and anair-to-liquid heat exchange assembly supported by the frame structure.The air-to-liquid heat exchange assembly includes an inlet coupled to acoolant supply line and an outlet coupled to a coolant return line forfacilitating passage of coolant therethrough. The air-to-liquid heatexchange assembly substantially covers the air outlet side of at leastone respective electronics rack, with air egressing therefrom passingthrough the air-to-liquid heat exchange assembly before being expelledinto the data center environment.

In a further aspect, a method of facilitating cooling of an electronicsrack is provided. The method includes: providing a cooling stationconfigured for disposition of an air outlet side of an electronics rackadjacent thereto for cooling egressing air from the electronics rack.When operational, air moves through the electronics rack from an airinlet side to an air outlet side thereof and then through the coolingstation disposed adjacent to the air outlet side of the electronicsrack. The cooling station is separate and freestanding from theelectronics rack, and includes: a frame structure separate andfreestanding from the electronics rack; an air-to-liquid heat exchangeassembly supported by the frame structure, the air-to-liquid heatexchange assembly including an inlet configured to couple to a coolantsupply line and an outlet configured to couple to a coolant return lineto facilitate passage of coolant therethrough; and wherein theair-to-liquid heat exchange assembly is sized to cool egressing air fromthe air outlet side of the electronics rack before being expelled into adata center.

Further, additional features and advantages are realized through thetechniques of the present invention. Other embodiments and aspects ofthe invention are described in detail herein and are considered a partof the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a partial embodiment of a layout of a liquid-cooled datacenter employing a cooling system, in accordance with an aspect of thepresent invention;

FIG. 3 depicts the liquid-cooled data center of FIG. 2 with oneembodiment of a plurality of frame structures of the cooling systemshown aligned in rows, in accordance with an aspect of the presentinvention;

FIG. 4 depicts a further embodiment of the frame structures of FIG. 3shown separate and freestanding within the data center, and illustratinga hinged flap attached to one edge thereof, in accordance with an aspectof the present invention;

FIG. 4A is a partial enlargement of the frame structure of FIG. 4, takenwithin circle 4A, and illustrating the hinged flap in rotation, inaccordance with an aspect of the present invention;

FIG. 5 depicts the liquid-cooled data center of FIG. 2, with a pluralityof frame structures as depicted in FIG. 4 disposed in multiple rowswithin the data center, in accordance with an aspect of the presentinvention;

FIG. 6A depicts installation of an air-to-liquid heat exchange assemblyonto one frame structure of FIG. 5, in accordance with an aspect of thepresent invention;

FIG. 6B illustrates one embodiment of the resultant cooling stationachieved by affixing the air-to-liquid heat exchange assembly to theframe structure in FIG. 6A, in accordance with an aspect of the presentinvention;

FIG. 6C depicts the cooling station of FIG. 6B showing the air-to-liquidheat exchange assembly pivoted open, in accordance with an aspect of thepresent invention;

FIG. 7A is a cross-sectional plan view of one embodiment of a coolingstation for a liquid-cooled data center, in accordance with an aspect ofthe present invention;

FIG. 7B is cross-sectional elevational view of the cooling station ofFIG. 7A, taken along line 7B-7B, in accordance with an aspect of thepresent invention;

FIG. 7C is a cross-section elevational view of the cooling station ofFIG. 7A, taken along line 7C-7C, in accordance with an aspect of thepresent invention;

FIG. 8 depicts the liquid-cooled data center of FIG. 5 with a pluralityof cooling stations disposed in multiple rows within the data center, inaccordance with an aspect of the present invention;

FIG. 9 depicts the liquid-cooled data center of FIG. 8 after a pluralityof electronics racks have been positioned therein, with an air outletside of each electronics rack disposed adjacent to a respective coolingstation of the cooling system, in accordance with an aspect of thepresent invention;

FIG. 10 depicts the liquid-cooled data center of FIG. 9, showingair-to-liquid heat exchange assemblies of multiple cooling stationspivoted open to allow access to the air outlet sides of respectiveelectronics racks, in accordance with an aspect of the presentinvention;

FIG. 11 depicts, for comparison against the raised floor layout of theair-cooled data center of FIG. 1, one embodiment of a raised floorlayout of a liquid-cooled data center employing multiple coolingstations to cool ten electronics racks, in accordance with an aspect ofthe present invention;

FIG. 12A partially depicts the data center layout of FIG. 10 duringassembly thereof, wherein guide rail assemblies are attached to thefloor adjacent to the cooling stations to facilitate disposition of theelectronics racks in position relative to the cooling stations, inaccordance with an aspect of the present invention;

FIG. 12B is an enlarged view of one embodiment of a guide rail assemblyfor directing and capturing one wheeled edge of an electronics rack whenbeing positioned in front of a respective cooling station, in accordancewith an aspect of the present invention;

FIG. 12C depicts the data center layout of FIG. 12A showing theelectronics racks being moved into position in front of the respectivecooling stations employing the guide rail assemblies, in accordance withan aspect of the present invention;

FIG. 12D depicts the data center layout of FIG. 12C, with theelectronics racks retained in position relative to the cooling stationsemploying the guide rail assemblies, in accordance with an aspect of thepresent invention; and

FIG. 13 depicts an enhanced embodiment of the liquid-cooled data centerof FIG. 2, wherein booster pumping units are utilized to facilitatemovement of coolant through the secondary supply and return headers, inaccordance with an aspect of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and include anyhousing, frame, rack, compartment, blade server system, etc., having oneor more heat generating components of a computer system or electronicssystem, and may be, for example, a stand alone computer processor havinghigh, mid or low end processing capability. In one embodiment, anelectronics rack may comprise multiple electronics drawers each havingone or more heat generating components disposed therein requiringcooling. Further, as used herein, “air-to-liquid heat exchange assembly”means any heat exchange mechanism characterized as described hereinthrough which liquid coolant can circulate; and includes, one or morediscrete air-to-liquid heat exchangers coupled either in series or inparallel. An air-to-liquid heat exchanger may comprise, for example, oneor more coolant flow paths, formed of thermally conductive tubing (suchas copper or other tubing) in thermal or mechanical contact with aplurality of air-cooled cooling fins. Additionally, size, configurationand construction of the air-to-liquid heat exchange assembly and/orair-to-liquid heat exchanger thereof as described herein below can varywithout departing from the scope of the present invention. Further,“data center” refers to a computer installation containing one or moreelectronics racks to be cooled. As a specific example, a data center mayinclude one or more rows of rack-mounted computing units, such as serverunits.

Reference is made below to the drawings, which are not drawn to scaleand are simplified for ease of understanding, wherein the same referencenumbers used throughout different figures designate the same or similarcomponents.

As noted initially, advances in semiconductor technology have led to anexponential increase in processor performance over the last few years.This in turn has led to a steep increase in the node, rack and clusterpower consumption, leading to a corresponding rise in the energy needsof the air-cooling HVAC equipment used for thermal management in atypical data center. These heat dissipation trends have also resulted insignificant cooling challenges due to localized hot spots in highdensity data centers.

FIG. 1 illustrates one embodiment of an air-cooled data center,generally denoted 100, wherein a plurality of electronics racks 110 aredisposed in multiple rows on a raised floor region 115 of the datacenter. Within air-conditioning units 120, air-to-liquid heat exchangerscool the room's ambient air transferring heat therefrom to facilitycoolant passing through the air-conditioning units 120, which are shownconnected to facility coolant inlet line 121 and facility coolant outletline 122. In data center 100, electronics rack 110 are aligned in rowswith air inlet sides 112 of the racks disposed adjacent to perforatedtiles 126, which allow cool air to be drawn into the inlet sides of theelectronics racks from an under-floor plenum 125 of raised floor region115. Heat is exhausted out the air outlet sides 114 of the electronicsracks into the data center environment. Typically, the electronics racksinclude one or more air moving devices which facilitate the ingress andegress of air flow from the air inlet side to the air outlet sidethereof. Heat expelled into the data center environment exits throughthe air-conditioning units 120.

The limiting factors for air-cooled data centers relate to the maximumchilled air that can be supplied from a single perforated tile, themaximum cooling capabilities of each air-conditioning unit, and a hotair recirculation phenomenon that is common in these systems. Hot airrecirculation occurs when the total supplied chilled air at the rackinlet is less than the total rack airflow rate, leading to the hotexhaust air from one electronics rack being drawn into the air inletside of the same or another electronics rack, thus resulting inunacceptably high air inlet temperatures.

The re-circulation of hot exhaust air from the hot aisle of the computerroom installation to the cold aisle can be detrimental to theperformance and reliability of the computer system(s) or electronicsystem(s) within the racks. Data center equipment is typically designedto operate with rack air inlet temperatures in the 10-35° C. range. Fora raised floor layout such as depicted in FIG. 1, however, temperaturescan range from 15-20° C. at the lower portion of the rack, close to thecooled air input floor vents, to as much as 45-50° C. at the upperportion of the electronics rack, due to hot air recirculation.

Thus, the state-of-the-art air-cooled data centers may have a number ofdisadvantages, which are to be addressed by the concepts presentedherein. These disadvantages include a moderate allowable server density,and therefore a lower allowable floor heat flux; a high energyconsumption; high space requirements, and thus higher total cost ofownership; high noise generation due to the power blowers in thecomputer room air-conditioning units being disposed within the datacenter; and varying electronic reliability due to uneven intaketemperatures.

FIG. 2 depicts one embodiment of a liquid-cooled data center (inaccordance with an aspect of the present invention), shown without theelectronics racks and cooling stations to cool the electronics racks.One or more coolant distribution units (CDUs) 210 reside on a floor 205,which again may be a raised floor region of data center 200. Eachcoolant distribution unit 210 includes at least one liquid-to-liquidheat exchanger which transfers heat from a system side to a facilityside thereof. As used herein, “facility water” or “facility coolant”refers to the facility water or coolant, while “system water” or “systemcoolant” refers to the cooled/conditioned water or coolant,respectively, circulating between the coolant distribution units and therespective cooling stations (described below). One example of facilityside or system side coolant is water. However, the concepts disclosedherein are readily adapted to use with other types of coolant on boththe facility side and the system side. For example, either coolant maycomprise a brine, a fluorocarbon liquid, or other similar chemicalcoolant or a refrigerant, while still maintaining the advantages andunique features of the present invention. In an alternate embodiment ofa liquid-cooled data center, no coolant distribution units are employed.In this embodiment, the system coolant is the facility coolant, which isfed directly to the cooling stations of the liquid-cooled data center,for example, through the supply and return headers discussed below withreference to the liquid-cooled data center embodiment of FIG. 2.

In addition to coolant distribution units 210, the cooling system forthe liquid-cooled data center of FIG. 2 includes a primary supply header213 and a primary return header 214, as well as multiple secondarysupply headers 215 and secondary return headers 216 extending from theprimary headers. As shown in the depicted embodiment, these headers aredisposed within the floor plenum 206 defined below raised floor 205 ofdata center 200. Quick connect couplings 217 are provided for connectingindividual cooling stations to the secondary supply and return headers215, 216. The quick connect couplings 217 define a relatively dense gridof coupling connections mounted to the secondary supply and returnheaders, through which system water is supplied and returned.

The three coolant distribution units (CDUs) 210 illustrated by FIG. 2are presented by way of example only. In certain implementations, oneCDU may be employed, while in others two or more than three may beutilized. As noted, the coolant distribution unit includes aliquid-to-liquid heat exchanger, which in one example, receives 5-10° C.facility water from a facility chiller plant (not shown) and cools thesystem water to a temperature in the range of 12-20° C., in order toremain above the room dew point temperature. In addition to aliquid-to-liquid heat exchanger, each CDU may include one or more of apower/control element, a reservoir/expansion tank, a pump, facilitywater inlet and outlet pipes 211, 212, a system coolant supply plenumand a system coolant return plenum. Further, each CDU may be designedwith multiple coolant pumps for redundancy, as well as moresophisticated control algorithms.

FIG. 3 depicts the liquid-cooled data center of FIG. 2 shown with theaddition of a plurality of frame structures 300 secured to the raisedfloor 205 in multiple rows 310. In one embodiment, frame structures 300are fabricated of steel and are rigidly bolted to the raised floor andto each other to form the illustrated cooling station superstructures.In the embodiment, adjacent to each frame structure 300 is an opening305 in raised floor 205 to facilitate plumbing and cable connections,for example, to the respective electronics rack and cooling station(described further below). Openings 305 allow the air-to-liquid heatexchange assemblies of the cooling stations to be fluidically coupled tothe secondary supply and return headers depicted in FIG. 2.

FIG. 4 illustrates one embodiment of an augmented frame structure 400 toform a portion of a cooling station of a cooling system as describedherein. As shown, this frame structure is augmented with a hinged flap401 on one edge thereof which is to facilitate the hinged attachment ofan air-to-liquid heat exchange assembly thereto. FIG. 4A depictsrotation of this hinged flap 401 relative to frame structure 400. Asshown in FIG. 4, frame structure 400 is bolted 402 to raised floor 205,as well as to one or more adjacent frame structures (such as shown inFIG. 3). The frame structure is further augmented by the addition ofstruts 403 which are also attached to raised floor 205 to enhancestrength and rigidity of the freestanding frame structure. In analternate embodiment, the frame structure may be augmented by theaddition of struts secured to a ceiling of the data center, oralternatively, to one or more walls of the data center.

FIG. 5 depicts the liquid-cooled data center of FIG. 3, but with theaugmented frame structure 400 of FIGS. 4 & 4A propagated throughout therows of support structures. Note that in this embodiment, each augmentedframe structure 400 is again shown bolted to raised floor 205 with anopening 305 disposed adjacent thereto for facilitating plumbing andcabling connections. Struts 403 of the augmented frame structures 400(in this embodiment) are shown shared between adjoining framestructures. Note also that in other implementations, one or more ofstruts 403 may be removed if unneeded for the rigidity and strength ofthe superframe structure(s) desired.

FIGS. 6A-6C illustrate assembly of a cooling station, in accordance withan aspect of the present invention. As shown in FIG. 6A, anair-to-liquid heat exchange assembly 610 is sized and configured tomount to hinged flap 401 (FIG. 4A) of augmented frame structure 400.FIG. 6B illustrates the assembled cooling station 600, including framestructure 400 and air-to-liquid heat exchange assembly 610 supported bythe frame structure. In FIG. 6C, air-to-liquid heat exchange assembly610 of cooling station 600 is rotated open, for example, to access theair outlet side of an adjacent electronics rack (as shown in FIG. 10).Note that in this embodiment, air-to-liquid heat exchange assembly 610is sized to substantially cover the air outlet side of at least onerespective electronics rack (see FIG. 9), wherein substantially all airegressing the air outlet side thereof passes through the air-to-liquidheat exchange assembly for extraction of heat before being expelled intothe data center room. The one-to-one correlation of cooling station toelectronics rack (e.g., illustrated in FIG. 9) is provided by way ofexample, only. In another implementation, multiple cooling stations maybe employed to cover the air outlet side of a single respectiveelectronics rack, or still further, one cooling station may cover theair outlet side of more than one respective electronics rack.

FIGS. 7A-7C depict in greater detail one embodiment of a cooling station600, in accordance with an aspect of the present invention. FIG. 7A is across-sectional plan view of cooling station 600, taken intermediate thetop and bottom thereof. As illustrated, flap 401 is hingedly mounted 705to frame structure 400. A plate 700 is attached (for example, by brazingor soldering) between an inlet plenum 701 and an outlet plenum 702 ofthe air-to-liquid heat exchange assembly 610. Inlet plenum 701 andoutlet plenum 702 extend vertically within the cooling station andmultiple horizontally-oriented heat exchange tube sections 710 areprovided, each having a coolant channel with an inlet coupled to theinlet plenum and an outlet channel coupled to the outlet plenum. Aplurality of vertically-extending rectangular fins 720 are attached tothe horizontally-oriented heat exchange tube sections for facilitatingtransfer of heat from airflow 705 passing through the air-to-liquid heatexchange assembly to system coolant flowing within the heat exchanger.System coolant introduced through inlet plenum 701 initially flowsthrough a front, cooler side of the horizontally-oriented heat exchangetube sections disposed closest to the air outlet side of one or morerespective electronics racks. The system coolant gains heat as it passesthrough each U-shaped tube section, until being expelled through outletplenum 702 on the back, warmer side of the tube sections.

FIG. 7B illustrates a cross-sectional elevational view of coolingstation 600 of FIG. 7A, taken along line 7B-7B, and with the coolingstation shown disposed on a floor through which flexible tubing 731, 732passes. The cooling station again includes frame structure 400 with ahinged flap 401 to which the air-to-liquid heat exchange assembly 610attaches. The vertically-oriented inlet plenum 701 is shown along with aplurality of horizontally-extending heat exchange tube sections 710 influidic communication therewith. Additionally, the vertically-alignedrectangular fins 720 are shown intermediate the tube sections in thiscross-sectional view. Inlet plenum 701 includes a Schrader valve 703 atan upper end thereof to allow purging of air from the inlet plenum, andis coupled via tubing 731 and quick connect couplings 730 to a secondarysupply header 215. Further, FIG. 7B illustrates tube 732, which isattached to a lower end of the outlet plenum, being connected via quickconnect couplings 730 to a secondary return header 216.

Similarly, FIG. 7C is a cross-sectional elevational view of coolingstation 600 of FIG. 7A, taken along line 7C-7C. In this view, thesecond, warmer side of the cooling station is illustrated. As shown,flap 401 is secured to hinge 705, which is attached to frame structure400. Plate 700 is attached to the heat exchanger between the inlet andoutlet plenums. Screws 740 hold plate 700 to hinged flap 401. Outletplenum 702 also includes a Schrader valve at the top thereof to purgeair therefrom, and a flexible tube 732 at a lower end thereof is coupledvia quick connect couplings 730 to a secondary return header 216. Inthis view, bolts 402 are illustrated fastening frame structure 400 to,for example, raised floor 205 of the data center.

FIG. 8 depicts one assembled embodiment of the cooling system disclosedherein. As illustrated, cooling stations 600 are aligned in multiplerows and secured to the data center floor 205. Openings 305 are disposedadjacent to each cooling station 600 to allow for plumbing access, asdescribed above in connection with FIGS. 7A-7C, as well as cable access,for example, for an electronics rack to be disposed adjacent to thecooling station. The flexible hoses and cables are routed through theopenings. With the array of cooling stations in place, a fully (in oneembodiment) liquid-cooled data center is enabled.

FIG. 9 illustrates the liquid-cooled data center of FIG. 8, withmultiple electronics racks 110, such as rack-mounted computer serverunits, disposed adjacent to the respective cooling stations 600 of thecooling system. The electronics racks are again air-cooled at the moduleand box level within the rack, and water-cooled at the rack level, thatis, by the cooling stations of the data center. In this embodiment, theair outlet side of each electronics rack is positioned adjacent to arespective cooling station to facilitate removal of heat egressing fromthe electronics rack. Air moving devices (e.g., fans or blowers) withinthe electronics racks force air across the electronics rack from an airinlet side to an air outlet side thereof, thus cooling the heatgenerating electronics within the rack. This hot exhaust air from theelectronics rack then passes through the respective air-to-liquid heatexchange assembly of the respective cooling station. The hot exhaust airis cooled by the cooling station and heat is rejected into the systemcoolant passing therethrough. The warm system coolant from theabove-floor air-to-liquid heat exchange assemblies is returned to therespective coolant distribution units (CDUs) via the secondary andprimary coolant headers described above. The system coolant is in turncooled within the CDUs in liquid-to-liquid heat exchangers by thechilled facility coolant. The cooled system coolant is then returned viathe supply headers to the cooling stations to continue the coolingcircuit.

The air-to-liquid heat exchange assemblies of the cooling stationsdepicted in FIG. 9 can be thermally designed to extract, for example,one hundred percent of the rack heat load, while imposing only a smallincrease (˜5-10%) in the air side impedance in the path of the rack airmoving devices. In the data center design of FIG. 9, the rack air movingdevices function to move air through both the server boxes (or nodes),as well as the adjacent cooling stations. Due to the small flowimpedance burden imposed by a cooling station, the adjacent electronicsrack can either absorb this penalty, or be designed to handle theadditional pressure drop associated with the cooling station. Also,those skilled in the art should note that the liquid-cooled data centerdesign of FIG. 9 does not require any conventional air-conditioningunit, thus eliminating one of the primary acoustic noise sources fromthe data center.

FIG. 10 illustrates the liquid-cooled data center of FIG. 9 with severalair-to-liquid heat exchange assemblies of the cooling stations 600pivoted open to access the air outlet sides 114 of the respectiveelectronics racks 110. This allows ready access to, for example, theelectronics within the racks from the air outlet sides thereof.

FIG. 11 depicts (for comparison) the same number of electronics racks110 disposed in two rows, as initially illustrated in the air-cooleddata center embodiment of FIG. 1. In the liquid-cooled data centerembodiment of FIG. 11, however, substantially smaller square footage isrequired. By way of example, and with reference to FIG. 1, consider a768 square foot air-cooled data region having ten high performance racksof 30 kW power dissipation each, and three 90-100 kW (30-ton)air-conditioning units supplying about 30,000 cubic feet per minute(CFM) of chilled air at ˜15° C. via 36 perforated tiles. Such a clusterhas a floor heat flux of 390 watts per square foot, which somewhatapproaches the air-cooling limit with respect to server density, andthus floor heat flux. Each of the 30-ton air-conditioning units consumesroughly 6.7 kW, thus making the total cooling power consumptionapproximately 20 kW for a 300 kW total server power. This translates toa modest coefficient of performance (COP) of 15. The COP is defined asthe ratio of the heat load cooled by the cooling energy consumed.

In the liquid-cooled data center embodiment of FIG. 11, the same tenhigh performance electronics racks, dissipating 30 kW each, are assumed.The three air-conditioning units in the data center unit of FIG. 1 arereplaced by one 12 kW coolant distribution unit. This translates to anexcellent COP of 25. The floor space is also reduced by a factor ofthree, thus reducing the cost while also enabling a much higher serverdensity. As noted, because of the removal of the air-conditioning units,the design of FIG. 11 will also be much quieter than the data center ofFIG. 1. As a further comparison, the cooling cost of ownership (CCO) forthe data center of FIG. 1 can be compared against the CCO for the datacenter of FIG. 11. The CCO is defined as the sum of the equipment costand the energy operating costs for a fixed lifecycle. By way of example,a three year time period is assumed, with an energy cost of 0.1 $/kWh.Total cost of ownership (TCO) would be a combination of the CCO, thefloor space cost, and several other components. It is expected while theCCO for both designs is nearly the same in the example provided, the TCOfor the liquid-cooled data center design will be much lower in view ofthe considerably less floor space employed. Thus, for the same CCO asthe air-cooled data center of FIG. 1, the liquid-cooled data center ofFIG. 11 shows significant energy, server density and space improvementover the traditional design.

FIGS. 12A-12D depict an alternative embodiment of a cooling system for aliquid-cooled data center, in accordance with an aspect of the presentinvention. In this embodiment, the cooling stations 600′, and inparticular, the air-to-liquid heat exchangers assemblies, are assumed tobe fixedly secured to the frame structures and hard-plumbed to thecoolant supply and return headers. This configuration might be requiredwhen using a refrigerant as the coolant, for example, in a facility thatdoes not allow water coolant to be employed. Alternatively, hardplumbing may be preferred by certain customers for reliability reasons.Hard plumbing of the air-to-liquid heat exchange assemblies to theunder-floor supply and return headers means that the heat exchangers cannot swing open to allow access or servicing at the air outlet side ofthe adjacent electronics racks since flexible hoses have been replacedby rigid pipes in this design.

In FIG. 12A, guide rail assemblies 1200 are shown affixed to the floor.These guide rail assemblies engage wheels of the electronics racks tofacilitate positioning of the electronics rack in alignment with one ormore of the respective cooling stations. A detailed view of one guiderail assembly 1200 is illustrated in FIG. 12B. As shown, each railassembly includes two guiding rails 1202 that are bolted 1203 to thefloor and are connected by two hinged flaps 1204, 1206, which also actas stoppers. With the rear stopper 1204 closed and the front stopper1206 open, the electronics racks are rolled towards the set of railassemblies until the rack wheels for each rack enter the two pathscreated by a pair of guide rails assemblies disposed in front of therespective cooling station. Once engaged, the electronics racks rollinto position being guided by the rail assemblies. After moving theelectronics racks into correct position, the front stoppers are locked,thus trapping the respective electronics racks in their proper location.The stoppers 1204, 1206 can be spring-loaded mechanisms or be manuallyclosed.

FIG. 12C illustrates the electronics racks 110 being moved into positionwith the air outlet sides 114 thereof disposed facing the coolingstations 600′. Proper alignment of the electronics racks to the coolingstations is achieved using the guide rail assemblies. FIG. 12Dillustrates the fully engaged position where the electronics racks 110are disposed adjacent to the cooling stations 600′.

The primary and secondary system coolant headers may form a complicatedflow passage for the coolant. This has the potential to create animbalanced flow distribution throughout the coolant grid under the floorand result in a large variation in system coolant flow rate. This inturn can lead to fluctuation and unpredictability in the thermalperformance of the heat exchangers. To address this issue, boosterpumping units (BPUs) can be employed as illustrated in FIG. 13. Each BPU1300 may include one or more coolant pumps, a reservoir tank, a filter,and related control electronics. While these BPUs can address poor flowdistribution, they can also potentially be employed to force differentcoolant flow rates in different parts of the data center. Thus, it ispossible to intentionally control and manage system coolant flow ratesin different parts of the liquid-cooled data center. The BPUs 1300 canhave flexible hoses at the four ports with quick connect couplings, ormay be hard plumbed into strategic locations on the data center floor.Further, the control instrumentation for these BPUs can be such thatthey allow a data center operator to remotely change the controlparameters and manage the data center system coolant flow ratesefficiently.

Although preferred embodiments have been depicted and described indetail herein, it will be apparent to those skilled in the relevant artthat various modifications, additions, substitutions and the like can bemade without departing from the spirit of the invention and these aretherefore considered to be within the scope of the invention as definedin the following claims.

1-16. (canceled)
 17. A method of facilitating cooling of an electronicsrack, the method comprising: providing a cooling station configured fordisposition of an air outlet side of an electronics rack adjacentthereto for cooling egressing air from the electronics rack, whereinwhen operational, air moves through the electronics rack from an airinlet side to the air outlet side thereof and then through the coolingstation disposed adjacent to the air outlet side of the electronicsrack, and wherein the cooling station is separate and freestanding fromthe electronics rack, and further the cooling station comprises: a framestructure separate and freestanding from the electronics rack; anair-to-liquid heat exchange assembly supported by the frame structure,the air-to-liquid heat exchange assembly including an inlet configuredto couple to a coolant supply line and an outlet configured to couple toa coolant return line to facilitate passage of coolant therethrough;wherein the air-to-liquid heat exchange assembly is sized to cool airegressing from the air outlet side of the electronics rack before beingexpelled into a data center; and wherein the air-to-liquid heat exchangeassembly is hingedly mounted to the frame structure to allow for accessto the air outlet side of the electronics rack when the electronics rackis disposed in operative position with the air outlet side thereofadjacent to the cooling station for cooling egressing air therefrombefore being expelled into the data center.
 18. The method of claim 17,wherein the air-to-liquid heat exchange assembly is sized tosubstantially cover the air outlet side of the electronics rack, andwhen the cooling station is operational, substantially all air egressingfrom the air outlet side of the electronics rack passes through theair-to-liquid heat exchange assembly for extraction of heat therefrombefore being expelled into the data center.
 19. (canceled)
 20. Themethod of claim 17, further comprising providing at least one coolantdistribution unit for providing cooled system coolant to the coolingstation and expelling heat from system coolant returning from thecooling station to facility coolant of the data center, and providing aprimary supply header, and a secondary supply header for supplyingcooled system coolant to the cooling station from the at least onecoolant distribution unit, and providing a primary return header and asecondary return header for returning system coolant from the coolingstation to the at least one coolant distribution unit, and furthercomprising providing at least one booster pumping unit coupled to thesecondary supply and return headers for controlling system coolant flowrate therethrough.